1. Field of the Invention
The present invention relates to composition processing of a multi-viewpoint image.
2. Description of the Related Art
Conventionally, when an image was captured by erroneous focus adjustment of a camera, it was necessary to recapture the image after performing focus adjustment again. Further, when it was desired to obtain focused images of a plurality of subjects the depths of which are different from each other, it was necessary to capture the image of each subject in focus a plurality of times.
In recent years, a technique called light field photography has been developed, that is capable of acquiring images from multiple viewpoints by adding a new optical element to the optical system and of adjusting the focus position by the later image processing (refocus).
By using this technique, there is an advantage that a failure in focus adjustment at the time of image capturing can be recovered by image processing because focus adjustment can be performed after image capturing. Further, there is also an advantage that it is possible to acquire a plurality of images focused on arbitrary subjects in an image from one captured image by changing the image processing method, and therefore, it is possible to reduce the number of times of image capturing.
In light field photography, the direction and intensity of a light beam that passes through each position (light field, hereinafter, referred to as “LF”) in a plurality of positions in the space are calculated from multi-viewpoint image data. Then, by using the information of the obtained LF, an image on the assumption that light passes through a virtual optical system and forms the image on a virtual sensor is calculated. By appropriately setting such a virtual optical system and a virtual sensor, refocus described previously is enabled. As an image capturing device for obtaining LF, a Plenoptic camera in which a microlens array is placed behind a main lens and a camera array in which compact cameras are arranged side by side are known. It is possible for both to acquire a multi-viewpoint image in which the image of a subject is captured in different directions by one-time image capturing. It is also possible to represent light field photography as calculation of an image acquired by a virtual sensor under virtual optical conditions from multi-viewpoint image data. In the following, the processing to calculate an image acquired by a virtual sensor is referred to as “refocus processing”. As refocus processing, there is known a method in which acquired multi-viewpoint image data is subjected to projective transformation onto a virtual sensor, and added and averaged (WO 2008/050904).
In such refocus processing, the value of a pixel on a virtual sensor is calculated using a pixel of a multi-viewpoint image corresponding to the position of the pixel. Normally, to one pixel of a virtual sensor, a plurality of pixels of a multi-viewpoint image corresponds.
In WO 2008/050904 described above, a method of refocus processing of a color image is not described, but, it is possible to easily infer that refocus processing of a color image can be performed by performing processing separately for each of RGB planes.
A case is considered, where a sensor that acquires multi-viewpoint image data is a sensor that acquires a color by a color filter array (CFA), such as a Bayer array. In this case, when refocus processing is performed for each color plane described above, processing to interpolate a color missing in a sensor output pixel is required before the refocus processing. However, if color interpolation processing is performed, each part of the multi-viewpoint image (hereinafter, referred to as a “sub image”) is blurred, and therefore, sharpness is reduced. Then, as a result of image composition using sub images the sharpness of which is reduced, blurring occurs also in a composite image, and therefore, the sharpness of the image acquired finally is also reduced.